Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace rclone mount of dcache with a Samba mount #162

Closed
wants to merge 22 commits into from
Closed

Replace rclone mount of dcache with a Samba mount #162

wants to merge 22 commits into from

Conversation

sverhoeven
Copy link
Member

@sverhoeven sverhoeven commented Apr 10, 2024 •
edited
Loading

sverhoeven and others added 13 commits April 10, 2024 15:22
@sverhoeven
Replace rclone mount of dcache with a Samba mount
c43c4b1 
Fixes #159
@sverhoeven
Improve spacing
9cec3c1
@sverhoeven
Creating playbook for filling file server
b20640d
@sverhoeven
Make linter happier
31b6553
@sverhoeven
Lower bar for linter in ci
655fbd1
@sverhoeven
Use Ansible/era5cli/esmvalcore to download ERA5
9a50fcb
@sverhoeven
Use scale/offset and zlib to convert era5 raw files to esmvaltool com…
ab79fe9 
…patible files
@sverhoeven
Blocks cannot be looped task files can
6a99e64
@sverhoeven
Got it running + Temporary disable removal raw files
daecac5
@sverhoeven
Multi machine vagrantfile
f4e40d5 
One for jupyter and one for file sever
@sverhoeven
Got era5process.py to run, sadly killed due to out of memory
8df80d4 
Increased vagrant vm memory from 8Gb to 16Gb , but still need to try it.
Known to work on bare metal with 32Gb RAM
@sverhoeven
Clean up raw era5 files after cmorization
08316f2
@sverhoeven
SRC ships with acient ansible, upgrade it
79522a5
@sverhoeven
Copy link
Member Author

Running playbook on SRC file server with tmux.

@sverhoeven sverhoeven mentioned this pull request May 16, 2024
Closed
@sverhoeven
Copy link
Member Author

Got

fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/opt/conda//bin/python", "/tmp/era5process.py", "1990", "mean_sea_level_pressure", "psl", "/data/volume_2/samba-share/climate-data/obs6/Tier3/ERA5/OBS6_ERA5_reanaly_1_day_psl_1990-1990.nc"], "delta": "0:00:05.541362", "end": "2024-05-17 07:47:59.721476", "msg": "non-zero return code", "rc": 1, "start": "2024-05-17 07:47:54.180114", "stderr": "ERROR 1: PROJ: proj_create_from_database: Open of /opt/conda/share/proj failed
Traceback (most recent call last):
  File \"/tmp/era5process.py\", line 46, in <module>
    orig_file = eds.files[0]
                ~~~~~~~~~^^^
IndexError: list index out of range", "stderr_lines": ["ERROR 1: PROJ: proj_create_from_database: Open of /opt/conda/share/proj failed", "Traceback (most recent call last):", "  File \"/tmp/era5process.py\", line 46, in <module>", "    orig_file = eds.files[0]", "                ~~~~~~~~~^^^", "IndexError: list index out of range"], "stdout": "Processing ERA5 mean_sea_level_pressure data from year 1990 to psl and saving to /data/volume_2/samba-share/climate-data/obs6/Tier3/ERA5/OBS6_ERA5_reanaly_1_day_psl_1990-1990.nc", "stdout_lines": ["Processing ERA5 mean_sea_level_pressure data from year 1990 to psl and saving to /data/volume_2/samba-share/climate-data/obs6/Tier3/ERA5/OBS6_ERA5_reanaly_1_day_psl_1990-1990.nc"]}

It was looking for input files in 1hrPt instead of 1hr.
We need to make scripts mip aware. For now resumed download of raw era5 data, skippiing cmorization.

@sverhoeven
Copy link
Member Author

Tthe climate data download completed, now stuck on

grdc download 
TASK [grdc : GTN-R - monthlies] ************************************************************************************************************************************************************Thursday 06 June 2024  11:47:58 +0200 (0:00:00.520)       0:15:47.553 *********                                                                                                             failed: [localhost] (item={'range': '1931-1960', 'url': 'https://gemstat.bafg.de/KiWISGRDC/KiWIS?datasource=1&service=SOS&version=2.0&request=GetObservation&featureOfInterest=http://gemstat.bafg.de/stations/1104150,1104530,1159100,1160235,1160378,1160500,1160580,1160684,1160788,1160880,1255100,1257100,1259100,1445100,1732100,1733600&temporalFilter=phenomenonTime,1931-01-01T00:00:00.000Z/1960-12-31T00:00:00.000Z'}) => {"ansible_loop_var": "period", "changed": false, "dest": "/data/volume_2/samba-share/observation/grdc/monthlies/GTNR_1.1931-1960.wml2", "elapsed": 0, "msg": "Request failed", "period": {"range": "1931-1960", "url": "https://gemstat.bafg.de/KiWISGRDC/KiWIS?datasource=1&service=SOS&version=2.0&request=GetObservation&featureOfInterest=http://gemstat.bafg.de/stations/1104150,1104530,1159100,1160235,1160378,1160500,1160580,1160684,1160788,1160880,1255100,1257100,1259100,1445100,1732100,1733600&temporalFilter=phenomenonTime,1931-01-01T00:00:00.000Z/1960-12-31T00:00:00.000Z"}, "response": "HTTP Error 404: Not Found", "status_code": 404, "url": "https://gemstat.bafg.de/KiWISGRDC/KiWIS?datasource=1&service=SOS&version=2.0&request=GetObservation&featureOfInterest=http://gemstat.bafg.de/stations/1104150,1104530,1159100,1160235,1160378,1160500,1160580,1160684,1160788,1160880,1255100,1257100,1259100,1445100,1732100,1733600&temporalFilter=phenomenonTime,1931-01-01T00:00:00.000Z/1960-12-31T00:00:00.000Z"}

@BSchilperoort
Copy link
Member

Tthe climate data download completed, now stuck on
grdc download

Seems like the GRDC uses a different portal now. I'm not sure if we can still download data from there :/

@sverhoeven sverhoeven mentioned this pull request Sep 3, 2024
Open
@sverhoeven
Copy link
Member Author

Merged into #169

@sverhoeven sverhoeven closed this Sep 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sverhoeven @BSchilperoort